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ABSTRACT 

We consider the estimation of a sparse parameter vector from 
measurements corrupted by white Gaussian noise. Our focus 
is on unbiased estimation as a setting under which the diffi- 
culty of the problem can be quantified analytically. We show 
that there are infinitely many unbiased estimators but none 
of them has uniformly minimum mean-squared error. We 
then provide lower and upper bounds on the Barankin bound, 
which describes the performance achievable by unbiased esti- 
mators. These bounds are used to predict the threshold region 
of practical estimators. 

Index Terms — Unbiased estimation, sparsity, denoising, 
Cramer-Rao bound, Barankin bound 

1. INTRODUCTION 

We consider the estimation of a parameter vector Xo £ M. N 
which is known to be S-sparse, i.e., at most S > 1 of its en- 
tries are nonzero, where typically S-€lN. This will be written 
as xo eX s = {x£l N : ||x|| <5}, where ||x|| Q denotes the 
number of nonzero elements of x. Our focus is the frequen- 
tist estimation setting in which the parameter Xo is modeled 
as unknown but deterministic [1]. While the sparsity S is as- 
sumed known, the positions of the nonzero entries of Xo are 
unknown. The S-sparse vector xq is corrupted by white Gaus- 
sian noise with known variance a 2 , so that the observation is 

y = x + n, with x G X s , n - A/"(0, cr 2 I w ) . (1) 

This will be called the sparse signal-in-noise model (SSNM). 

In this paper, we are interested in unbiased estimators, i.e., 
estimators x(y) which satisfy E x {x(y)} = x for all x6 Xs. 
(The notation E x indicates that the expectation uses the distri- 
bution of y induced by the parameter value x.) The assump- 
tion of unbiasedness is commonly applied when analyzing es- 
timation performance [1,2]. Apart from being intuitively ap- 
pealing, unbiased estimation is known to be optimal when the 
signal-to-noise ratio (SNR) is high [1]. More practically, to 
analyze achievable performance, the class of estimators must 
be restricted in some way in order to exclude approaches such 
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as x(y) = xi, which yield zero error at the specific point 
x = xi but are evidently undesirable. Consequently, even 
though many estimators for the SSNM are biased [3], we 
adopt the commonly used unbiasedness assumption in order 
to gain insight into the achievable performance in this setting. 
Note that while xo is known to be S'-sparse, the estimates x 
we consider are not constrained to be S'-sparse. 

Our goal is to characterize the minimum mean-squared er- 
ror (MSE) achievable by unbiased estimators. This optimal 
MSE, known as the Barankin bound (BB) [4], cannot be cal- 
culated analytically. Instead, we prove that the BB is itself 
bounded above and below by simple, closed-form expres- 
sions. The upper bound is obtained by solving a constrained 
optimization problem, while the lower bound is based on the 
Hammersley-Chapman-Robbins technique [5]. In a numeri- 
cal study, we demonstrate the ability of our bounds to roughly 
predict the threshold SNR of practical estimators. 

A well-known lower bound on the BB is the Cramer-Rao 
bound (CRB), which was recently derived for the sparse set- 
ting [2,6]. To obtain the CRB, it is sufficient to assume that 
the estimator x(y) is unbiased only at a given parameter value 
Xo S Xs and in its local neighborhood. By contrast, in this pa- 
per we seek to characterize the MSE obtainable by estimators 
which are unbiased for all values x 6 Xs- One may hope 
to achieve a higher — hence, tighter — lower bound by adopt- 
ing this more restrictive constraint. Furthermore, the CRB is 
discontinuous in x [6], whereas the MSE of an estimator for 
the SSNM is always continuous [1]; this again suggests that 
it may be possible to find improved lower bounds. 

We note in passing that our results can be easily generalized 
to the sparse orthonormal signal model 

y = Hx + n , with x e X s , n - JV (0, a 2 l M ) , (2) 

where y € M. with M > N and the known matrix H € 
jniMxJV nas orthonormal columns, i.e., H T H = Ijv- How- 
ever, for simplicity of notation, we will continue to refer to 
the model (1). This setting arises, for example, in image de- 
noising based on the assumption of sparsity in the wavelet 
domain. 

2. UNBIASED ESTIMATION 

We will denote the set of unbiased estimators by 

U = (x(-) : E x {x(y)}=x for all x€* s }. 



Before analyzing bounds on the performance of unbiased esti- 
mators, we must ask whether such techniques exist at all. This 
is answered affirmatively in the following theorem. Due to 
space limitations, the proof of this and subsequent theorems 
is omitted and will appear in [7]. The MSEE x {||x(y) — x|||} 
of a given estimator x(-) at the parameter value x will be de- 
noted e(x; x). 

Theorem 1 For S — N in the SSNM (1), there is only one 
unbiased estimator of xo ( up to modifications having zero 
measure) which has a bounded MSE. This is the ordinary 
least-squares (LS) estimator, 
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whose MSE is e(x;xLs) = Ncr 2 . By contrast, for S < N, 
there are infinitely many unbiased estimators ofx . 

The assumption of a bounded MSE means that there exists a 
constant C such that e(x; x) < C for all x. This assumption 
is required to exclude certain pathological functions which 
do not make sense as estimators. Since for S = N the only 
unbiased estimator with bounded MSE is the LS estimator, 
we will hereafter assume that S<N. 

3. OPTIMAL UNBIASED ESTIMATION 

Given a fixed parameter value x € Xs, one can define an 
optimal unbiased estimator x( x °)(-) for the SSNM (1) as an 
unbiased estimator which minimizes the MSE at x , i.e., 

e(x ;x( x °)) = min E xo {||x(y)-xo||^}. (4) 

Solving (4) is equivalent to solving the N individual opti- 
mization problems 

min E x J\x k (y) - x . k \ 2 } , k = 1, N , (5) 
x k (-)ew 

where and xo,fc are the fcth elements of x(-) and xo, 

respectively, and U 1 denotes the set of unbiased estimators of 
the fcth element of x. 

Since the MSE e(x; x) of an estimator x(-) depends on the 
true parameter value x<G Xs, an unbiased estimator that has a 
small MSE for one parameter value xo <G X$ may have a large 
MSE for other parameter values in X$. Sometimes, however, 
a single estimator solves (4) simultaneously for all x e X$. 
Such a technique is called a uniformly minimum variance un- 
biased (UMVU) estimator [1]. (Note that for an unbiased es- 
timator the variance equals the MSE, so minimum MSE im- 
plies minimum variance.) However, for the SSNM (1), the 
following negative result can be shown [7]. 

Theorem 2 For the SSNM with S<N, there does not exist a 
UMVU estimator. 

Thus, no single unbiased estimator achieves minimum 
MSE for all values of xG X$. We can still try to find a locally 
minimum variance unbiased (LMVU) estimator by solving 



(4) for a specific xo € Xs- This approach yields the unbiased 
estimator which has minimum MSE for the given parameter 
value x . While the resulting technique may have poor per- 
formance for other parameter values, its MSE at xo is of in- 
terest as it provides a lower bound on the MSE at xo of any 
unbiased estimator. This lower bound is known as the BB [4]. 
Because it is achieved, by some technique, at each point x , 
the BB is the largest (thus, tightest) lower bound for unbiased 
estimators. Unfortunately, an analytical expression of the BB 
is usually not obtainable. Instead, in the following two sub- 
sections, we will provide lower and upper bounds on the BB. 

3.1. Lower Bound on the MSE 

A variety of techniques exist for developing lower bounds on 
the MSE of unbiased estimators. The simplest of these is the 
CRB, which was derived for a more general sparse estimation 
setting in [2, 6]. In our case, the CRB is given by 



e(xo;x)> 



ll x o|| = s 
||x || < S. 



(6) 



The CRB assumes unbiasedness only in a neighborhood of 
xo. Since we are interested in estimators which are unbiased 
for all x <G Xs, we can expect to obtain higher bounds. One 
such approach is the Hammersley-Chapman-Robbins bound 
(HCRB) [5], which can be formulated, in our context, as fol- 
lows. For a given parameter value x , consider a set of p "test 
points" {vi}^ =1 such that + x € X$- The HCRB states 
that the MSE of any unbiased estimator x(-) £ U satisfies 

e(x ;x) >Tr(VjtV T ), 

where V = [vi • • • v p ] e R A ' xp , the elements of the matrix 
J € W x p are given by (J), M = exp(vf v^/cr 2 ) - 1, and Jt 
denotes the pseudoinverse of J. 

Both the test points Vj and their number p are arbitrary and 
can depend on xo. Clearly, the challenge is to choose test 
points which result in a tight but analytically tractable bound. 
To this end, we note the following facts. First, it can be shown 
[7] that the CRB is obtained as a limit of HCRBs by choosing 
the test points Vj as 1 

{*ei} ieBupp(xo) , if||xo|| = S (7a) 

{*ei}ie{i,...,iv}. if|l*o|| <S (7b) 

and letting t — > 0. Here, represents the ith column of the 
NxN identity matrix and supp(xo) denotes the set of indices 
of all nonzero elements of Xo. Second, the CRB (6) is tight 
whenever ||x || < S, since in this case the bound is achieved 
by the LS estimator (3). The CRB is also tight when ||xo|| = 
S and all components of x are much larger (in magnitude) 
than g [6]. Any laxity in the CRB will therefore arise when 
xo contains exactly S nonzero components, one or more of 
which are small. For such x , it follows from (7a) that only 
S test points are employed; yet when the small components 



1 Hereafter, with a slight abuse of notation, the index i of v j is allowed to 
take on p different values from the set {1, . . . , TV}. 



of Xo are replaced with zero, then according to (7b), a much 
larger number N of test points is used. In light of this, one 
could attempt to improve the CRB by increasing the number 
of test points when ||x || = S. This should be done such that 
when the smallest component tends to zero, the test points 
converge to (7b), for which the CRB is tight. In doing so, one 
still needs to ensure that Vj + x € Xs- 

A reasonable choice which satisfies these requirements is 



tei, i e supp(xo) 

-x ( S) e k + tei, i£ supp(xo) 



(8) 



for 



1, . . . , N. Here, x is the 5-largest (in magnitude) 

entry of x (note that = if ||x || < S). The test points 
(7), which yield the CRB, are a subset of the choice of test 
points in (8). Consequently, it can be shown that the resulting 
bound will always be at least as tight as the CRB. 

In analogy to the CRB, a simple lower bound can be ob- 
tained from (8) by letting MO, Some rather tedious calcu- 
lations [7] yield the following result. 

Theorem 3 The MSE of any unbiased estimator x(-) G U is 
bounded below by 



e(x ;x) > 



'Sa 2 + (N-S-l)<7 2 e-ti/° 2 , ||x || = S 



^Na 2 , 

where £ = min feesupp ( Xo) \x ,k\- 



ll x o|| < s 



(9) 



Strictly speaking, this bound is obtained as a limit of 
HCRBs. However, for simplicity we will continue to refer to 
(9) as an HCRB. The bound is higher (tighter) than the CRB 
(6), except when the CRB itself is already tight. Furthermore, 
while both bounds are discontinuous in the transition between 
||x || = S and ||x || < 5*, the jump in (9) is much smaller 
than that in (6). This discontinuity can be eliminated alto- 
gether by adding more test points, but the resulting bound no 
longer has a simple closed form and can only be evaluated 
numerically. 

3.2. Upper Bound on LMVU Performance 

We next derive an upper bound on the BB by approximately 
solving (5) for k = 1, . . . , N. When k e supp(x ), it can be 
shown [7] that the solution of (5) is given by the fcth compo- 
nent of the LS estimator in (3), i.e., 



xk(y) = jjk- 



(10) 



It remains to consider k ^ supp(xo). For such k, let us de- 
note .ife(y) = yk + x 'k(y)- To obtain an upper bound on 
the solution of the optimization problem (5), we will restrict 
the set of allowed estimators x k (•) by requiring the correction 
component x' k (-) to satisfy the following two properties. 

• Odd symmetry with respect to index k and all indices in 
supp(x ): For all I e {k} U supp(x ), 



Independence with respect to all other indices: For all 

I 4- {k} U supp(xo), 



x' k (...,yi,...) = x' k (. . . ,0, . . .) . 



(12) 



We introduce these constraints (which are rather ad-hoc) be- 
cause under them a closed-form solution of (5) can be ob- 
tained. This constrained solution is given by [7] 



£c,fc(y;x ) = y k 



i- n -4*^ 

(Gsupp(xo) 



(13) 



for y e . For other values of y, the solution can be ob- 
tained via the properties (11) and (12). 

Combining (10) and (13), we obtain a constrained solu- 
tion of (4) which will be denoted x c (y;x ). The MSE of 
x c (y;x ) at the parameter value x is the tightest lower 
bound on the MSE at xo of any unbiased estimator that sat- 
isfies the constraints (11) and (12). It will thus be called the 
constrained BB and denoted by BB c (x ). Note that BB c (x ) 
is an upper bound on the BB. A closed-form expression of 
BB c (xq) is given in the next theorem. 

Theorem 4 The BB (i.e., the minimum MSE achievable by 
any unbiased estimator) at Xp is bounded above by 



BB c (x ) = Sa 2 + (N-S)a 2 
with 

g(x;a ) = 



i- J] g{xo,i;v 2 

ZGsupp(xo) 



f jf ~ -<y» + -W) S inh(gf) tanh(gf) dy . 



x' k (...,-yi,...) = -x' k (...,yi,...) 



(11) 



We note that for all parameter values x with ||x || < S, 
the lower and upper bounds of Theorem 3 and 4 coincide and 
equal Na 2 , which is therefore also the BB. This value is iden- 
tical to the unconstrained CRB. We refer to [2, 6] for a discus- 
sion of this result. 

4. NUMERICAL RESULTS 

In Section 3, we bounded the achievable performance of unbi- 
ased estimators as a means for quantifying the difficulty of es- 
timation in the SSNM. One use of this analysis is in the iden- 
tification of the threshold region, a range of SNR values which 
constitutes a transition between low-noise and high-noise be- 
havior. Specifically, the performance of estimators can often 
be calculated analytically when the SNR ||xo|||/ (Na 2 ) is ei- 
ther very high or very low. It is then important to identify the 
threshold which separates these two regimes. 

The lower and upper bounds on the BB which were derived 
above also exhibit a transition between a high-SNR region 
and a low-SNR region. In the high-SNR region, the lower 
bound (assuming ||x || = S) and the upper bound both con- 
verge to Sa 2 , while in the low-SNR region, both bounds are 
on the order of Na 2 . The true BB therefore also displays such 
a threshold. Since the BB is itself a lower bound on the MSE 
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Fig. 1. MSE of the HT and ML estimators compared with the 
performance bounds BB C , HCRB and CRB (all normalized 
by a 2 ), as a function of the SNR. 

of unbiased estimators, one would expect that the transition 
region of the BB occurs at slightly lower SNR values than 
that of actual estimators. 

To test this hypothesis, we compared the bounds of Sec- 
tion 3 with the MSE of two well-known biased estimation 
schemes. First, we considered the maximum likelihood (ML) 
estimator, which can be shown to equal 



^ML,fc(y) 



y k , ifkeC 
0, else 



(14) 



where C denotes the indices of the S largest (in magnitude) 
entries of y. We also considered the hard-thresholding (HT) 
estimator 

Vk, if \Vk\ > T 
0, else 



#HT,fc(y) 



(15) 



with the commonly employed threshold T = o^/2\ogN [3]. 

We used N = 10 and S = 4, and chose 50 parameter vec- 
tors x from the set ft = {c (1, 1, 1, 1, 0, 0, 0, 0, 0, 0) T } ceR+ , 
with different values of c to obtain a wide range of SNR val- 
ues. The results are plotted in Fig. 1. Although there is some 
gap between the lower bound (HCRB) and the upper bound 
(BB C ), a rough indication of the behavior of the BB is con- 
veyed. As expected, the SNR threshold predicted by these 
bounds is somewhat lower than that of practical estimators. 
Specifically, the transition region of the BB can be seen to oc- 
cur at SNR values between —5 and 5 dB, while the transition 
of the ML and HT estimators is at SNR values between and 
10 dB. Another effect which is visible in Fig. 1 is the con- 
vergence of the ML estimator to the BB at high SNR; this is 
a consequence of the well-known fact that the ML approach 
is asymptotically unbiased and asymptotically minimizes the 
MSE at high SNR. Furthermore, our performance bounds in 
Fig. 1 suggest that at intermediate and high SNR (above 
dB), there may exist unbiased estimators that outperform the 
ML and HT estimators. However, at low SNR, both estima- 
tors (ML and HT) are better than the best unbiased estimator. 
This agrees with the general rule that unbiased estimators per- 
form poorly at low SNR. 

One may argue that considering only parameter values 
in the set TZ is not representative, since TZ covers only a 
small part of the parameter space X$. However, it can be 
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Fig. 2. Ratio BB c (x )/HCRB(x ) for different sets of pa- 
rameter vectors Xq. 

shown that the choice for TZ is conservative in that the max- 
imum deviation between the HCRB and the BB C is largest 
when the non-zero entries of Xo have approximately the 
same magnitude, which is the case for each element of 
TZ. To illustrate this fact, we considered the two additional 
sets TZ 2 = {c(0.1,l,l,l,0,0,0,0,0,0) T } ceR+ and TZ 3 = 
{c (10, 1, 1, 1, 0, 0, 0, 0, 0, 0) T } ceR+ , in which the four non- 
zeros are not all equal. Fig. 2 depicts the ratio BB C /HCRB 
versus the SNR for the three sets TZ, TZ2, and TZ 3 . It is 
seen that the maximum value of BB C /HCRB is indeed highest 
when xo is in TZ. 

5. CONCLUSION 

We considered unbiased estimation of a sparse parameter vec- 
tor in white Gaussian noise. The Barankin bound (i.e., the 
minimum MSE achievable by any unbiased estimator) was 
characterized via upper and lower bounds. Our numerical 
results suggest that these bounds may also give information 
about the success of more general, biased techniques, for ex- 
ample by providing a lower bound on the threshold region. A 
subject for further study is the extension of these results to a 
model of the form y = Axo + n in which Xo is sparse and A 
is an arbitrary matrix with full column rank. 
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